Description

The v1 env-server pool (verifiers/v1/serve/pool.py) — the process that actually serves rollouts during training — has no observability surface today: no stats snapshot, no /metrics, no way to see how many rollouts are in flight or whether a worker is saturated. The only existing metrics work (#1415) targets the legacy v0 verifiers/serve/server/ architecture, so v1 runs blind.

This adds a small, dependency-free Prometheus endpoint for the v1 pool, addressing #1188.

What it does

• New verifiers/v1/serve/metrics.py: a hand-rendered Prometheus text renderer (PoolMetrics) plus a minimal asyncio.start_server-based MetricsServer exposing GET /metrics. No new dependencies.
• The broker loop in EnvServerPool.run() updates cheap, O(1) counters/gauges as it dispatches and reaps requests; the metrics server is started/stopped inside the existing try/finally so its lifecycle is coupled to the broker.
• Opt-in: metrics_port is unset by default (no port → no server). metrics_address defaults to 127.0.0.1 (loopback) rather than binding all interfaces. Both are threaded through ServeConfig and the serve CLI.
• Pool-only: single-server (non-pool) mode is unchanged and logs a warning if metrics are configured.

Metrics exposed

verifiers_v1_env_active_rollouts                      # aggregate in-flight slots
verifiers_v1_env_worker_active_rollouts{worker_index} # per-worker in-flight slots
verifiers_v1_env_workers                              # current worker processes
verifiers_v1_env_configured_workers                   # configured limit (0 = unbounded)
verifiers_v1_env_pending_requests                     # requests awaiting a worker reply
verifiers_v1_env_requests_total                       # dispatched requests (counter)
verifiers_v1_env_request_latency_seconds_{count,sum}  # dispatch->reply latency (summary)

Deliberately scoped to observability: no worker-restart or event-loop-lag gauges, since the pool has no restart controller or lag monitor yet (exposing those would be misleading, and lifecycle work is being handled separately in #1993/#1995). The reply path does no payload decoding, so the broker's zero-copy copy=False forward is preserved.

Type of Change

• New feature (non-breaking change which adds functionality)

Testing

• All existing tests pass when running uv run pytest locally (targeted: tests/v1/test_configs.py, tests/test_env_server.py — 26 passed).
• New tests have been added to cover the changes

Per AGENTS.md ("Do not add unit tests to your PRs"), no unit tests are added. Verified with a throwaway script that starts a real one-worker EnvServerPool, issues a request, and curls the endpoint:

HTTP/1.1 200 OK
Content-Type: text/plain; version=0.0.4; charset=utf-8
verifiers_v1_env_active_rollouts 0
verifiers_v1_env_worker_active_rollouts{worker_index="0"} 0
verifiers_v1_env_workers 1
verifiers_v1_env_configured_workers 1
verifiers_v1_env_pending_requests 0
verifiers_v1_env_requests_total 1
verifiers_v1_env_request_latency_seconds_count 1
verifiers_v1_env_request_latency_seconds_sum 1.157...

Checklist

• My code follows the style guidelines of this project as outlined in AGENTS.md
• I have performed a self-review of my own code
• My changes generate no new warnings

Additional Notes

No new dependencies. Endpoint is disabled unless metrics_port is set, and binds to loopback by default. Related: #1188 (request), #1415 (v0 equivalent).